1. Field of the Invention
The present invention relates generally to the field of software testing and specifically to evaluating how well a given computer program or set of programs have been tested.
2. Description of Backgound Art
Software developers commonly spend a great deal of time and money testing the computer programs they develop. This is especially true for development of so-called “mission- critical” programs, such as those used in business accounting, payroll and billing systems.
Oftentimes, managers, developers and testers are unable to verify how well software has been tested in order to predict the likelihood that untested problems may emerge at some later time. Though extensive testing may have been undertaken, significant portions of the program logic may not have been exercised.
Certain other solutions display the structure of a program to be tested and assist testers in identifying portions of the program that should be tested. For example, the McCABE TEST product provided by McCabe & Associates, Columbia, Md., is advertised as providing a visual environment showing graphical displays of test paths and visually identifying the most complex areas to help focus testing efforts. This product is further advertised as identifying tested and untested path and pinpointing problematic areas. The product is asserted to provide “multiple levels of code coverage including unit level, integration level, path coverage, branch coverage, and Boolean coverage.” Thus, this product is advertised to allow one to identify which parts of a system remain untested.
The XPEDITER debugging product, provided by Compuware Corporation of Farmington Hills, Mich., identifies which lines of a program have been executed, but does not in itself indicate how much of a program's logic has been executed, since different logic paths may invoke the same program lines under different logical circumstances. The XPEDITER product permits programmers and testers to watch source code execute as well as to control program execution by suspending executing, displaying and changing variables, altering logic, setting statement execution counts, and resuming execution from a desired point in the program. The CICS version of the XPEDITER product further includes a file utility for editing and browsing VSAM, temporary storage and transient data files for debugging applications running in CICS.
While testing every line of executable code in a program will be expected to uncover many errors in the program, it is insufficient since a line of code may execute properly under one set of conditions and not under another set of conditions. For example, consider a program represented in pseudo code as:
begin
b=input data;
if b>0, then b=b+3;
output=sqrt(b);
end.
If this program is tested with input data greater than zero, each program statement will execute, but only one of the two logical paths of the program will have been tested (i.e., the path with b>0). Such testing is insufficient, since the other logical path (i.e., the path with b≦0) will always result in an error caused by attempting to find the square root of a negative number.
Therefore, for the testing to be a reliable indicator of actual program performance, it is important to ensure not only that all statements of the program are executed, but that the tester be able to tell which of the logical paths of the program have been tested and which have not.
Such path testing can be helpful in uncovering common, but difficult to locate, bugs such as are caused by attempts to divide by zero and date errors caused by transitions from the years 19___ to 20___.
Some related issues are addressed in U.S. Pat. No. 4,853,851, which discloses a method of determining static results as well as which program paths have been executed, and representing non-executed statements and paths. However, this patent only discloses use of difference between two instruction addresses to determine whether or not a branch instruction has been executed, which limits applicability of this technique to low-level programming languages such as assembly language and microcode. Additionally, this technique calls for the use of special-purpose hardware to collect coverage results. Still farther, this disclosure does not specify the manner in which coverage results are reported.
Collection of execution data is also addressed in U.S. Pat. No. 5,050,168, but this reference does not address any manner of static analysis of program structure or reporting of path coverage.
What is needed is an improved system and method for testing computer programs that allows computer programmers and quality assurance specialists to (i) collect the results of their testing efforts and create an audit trail of program test execution; (ii) specify recording criteria to identify the percentage of an application that was exercised and the amount that it was exercised, determine what percentage of modified code in a program has been tested based on the tests already completed; and (iii) view graphical structure charts to determine requirements for additional test data and user input. In particular, it would be desirable for a system to take advantage of existing debugger and analyzer components already installed on a mainframe computer of a data processing system in order to achieve the operation described above.